There can be two answers
One involves math, one doesn't!
Answer 1: Only one person was going to St. Ives. The person telling the tale. Assuming that they met because they crossed paths and were going to opposite places. Answer 2: If you want to assume they were ALL going to St Ives, the answer is as such: 1 man telling the tale 1 man with 7 wives 7 wives 49 sacks 343 cats 2401 kittens = 2802 were going to St Ives