Moving human is quite complicated to track since there are variations in background, texture, and lighting in an environment. This paper presents an effective method for tracking a human guide from a camera follower in both indoor and outdoor condition. This algorithm is designed to be embedded in a smart wheelchair. A conventional human detection by using Histogram of Oriented Gradient (HOG) was used at the first stage, then each detected human by HOG is utilized for tracking algorithm. The detected area from HOG is converted to grayscale image and its Entropy Difference Minimization (HOG-EDM) is calculated. The process is repeated for every frame. The entropy minimization is used as matching function in the tracking subsystem to determine the candidate of tracked object in the upcoming frame. The proposed algorithm has been proven to work well in indoor and outdoor area, even with textured background. Our testing based on self-made and public datasets shows that HOG-EDM method reaches over 80% accuracy