I've come across this before,
Wood's map of 1834 shows our High Street as 4 Streets (1) High Street 2) Market Street 3) Castle Street 4) Castle Bank), so at least you know where the High Street was roughly. The fact that period predates any high resolution building maps or census really does make the job difficult.
There is stained glass numbering to the tansom/fanlights of some building such as 'SAKS' showing it as 205.
... SAKS (or the preceding building) was presumably the start of Market Street in 1834. So you even have to be careful with those!