||From I-5 take the Old Fairhaven Pkwy exit and after about 2 mi on that road turn left onto chuckanut drive. Follow chuckanut drive out for about 8 miles then turn right into Larabee State Park (well marked).
(From user valeberga): Please do not try to access these walls from Larrabee State Park proper. The railroad owns the property, and neither the rail company nor the state park rangers want to see you there. Besides, all access to the tracks from the park are fenced off, with prison-style barbed wire in places. Instead, ride you bike down there (interurban trail) or park in the Clayton Beach parking lot. To the south of the park, there are trails from the highway leading down to the cliffs. Stay away from the tracks at all times and try not to be conspicuous. You could be charged with tresspassing. I've had good luck down there, but I've also been lucky enough to avoid being confronted.
(Updated by user:naturalhigh)
Railroad people won't bother you as long as you stay clear of the tracks. DON'T SET ANYTHING on the tracks! valeberga is right, fencing has gone up in larrabee state park, restricting access to the areas since I wrote the original description. Clayton Beach parking area is now your best bet. Also, Larrabee state park now has a daily use fee, I believe.