Drivers’ behavior evaluation is one of the most important problems in intelligent transportation systems and driver assistant systems. It has a great influence on driving safety and fuel consumption. One of the challenges in this regard is the modeling perspective to treat with uncertainty in judgments about driving behaviors. Really, assessing a single maneuver with a rigid threshold leads to a weak judgment for driving evaluation. To fill this gap, a novel neuro-fuzzy system is proposed to classify the driving behaviors based on their similarities to fuzzy patterns when all of the various maneuvers are stated with some fuzzy numbers. These patterns are also fuzzy numbers and they are extracted from statistical analysis on the smartphone sensors data. Our driving evaluation system consists of three processes. Firstly, it detects the type of all of the maneuvers through the driving period, by using a multi-layer perceptron neural network. Secondly, it extracts a new feature based on the acceleration and assigns three fuzzy numbers to driver’s lane change, turn and U-turn maneuvers. Thirdly, it determines the similarity between these three fuzzy numbers and the fuzzy patterns to evaluate the safe and the aggressive driving scores. To validate this model, Driver’s Angry Score (DAS) questionnaires are used. Results show that the fusion of Inertial Measurement Unit (IMU) sensors of smartphones is enough for the proposed driving evaluation system. Accuracy of this system is 87% without using GPS and GIS data and this system is independent of smartphones and vehicles types.