Sign language is considered a spatial-temporal representation of patterns, where curvature, acceleration, and dynamic intensity play major roles. However, existing self-supervised skeleton-based sign language recognition (SLR) methods primarily depend on static joint configurations and first-order derivatives, limiting motion-capture from higher-order motion derivatives. In this paper, we propose an Attention-adaptive Second-order Motion (AaSoM) framework, a hybrid Attention-BiGRU architecture for Indian Sign Language (ISL) recognition that operates on three-stream skeletal kinematics (pose, velocity and acceleration), extracted using MediaPipe and concatenated to a unified spatial-temporal representation. Our proposed approach enriches skeletal representation and captures salient inter-joint dependencies that demonstrate robust temporal modeling with strong generalization across the datasets used. The performance of our AaSoM framework is evaluated using loss function, and accuracy metric. Our experimentation achieved a testing accuracy of 89.14% and reduced loss across 50 epochs. Along with this we have compared our framework with other state-of-the-art models. Thus our findings highlight the need for a higher-order dynamics (second-order) in skeleton-based (spatial-temporal) self-supervised SLR learning.