At first glance, the C15 'New Haven' may seem an unlikely place for pilgrims, but it replaced the medieval port of Seaford, which had a flourishing trade with France, and a hospital dedicated to St James (a dependency of the Cistercian abbey at Robertsbridge). Inland, along the river Ouse, was Lewes, where the large monastery, dedicated, unusually, to St Pancras, was the Cluniac HQ in England, and provided monks for Reading. It was built of Caen stone, brought, of course, by sea. Just N of Seaford is Bishopstone, 'estate of the bishop' (of Chichester), where the church has Saxon remnants.
Lewes was at the S end of a Roman road from London. Where this crosses the Pilgrims Way at Titsey, there was a Romano-British temple. Now there is a church dedicated to St James (though the dedication may well have been made when the church was moved by the landowner in Victorian times, and due to the proximity of the Pilgrims Way). To the W is Tandridge, where the Augustinian priory was dedicated to St James (the current church, outside which is an ancient yew, is St Peter's).
You can roughly follow the Roman road by taking the Wandle Trail from Wandsworth past Merton Priory to Croydon, then the Vanguard Way to Titsey and Uckfield, where you can cross to the Sussex Ouse Valley Way to Lewes and Newhaven.