5 of you huh?
I've never seen workers out there when I've been out there, but I showed up and they were around I would turn around an find another place to hang out.
Like Kartessa said, it's on private propety. Yes, lots of people hang out down there, far more kids and dog walkers than climbers, but that doesn't change the situation.
Also, there are what, three homes on that last little strech of road before the tracks? They don't seem to like people parking in front of their places. It's not hard to find parking a little further away that doesn't seem to bother anyone.