Typically, we will have the elastomer molded onto the core rather than using tube or such stretched over the core.  I am not certain that is what you are doing here, but we have found in this case that the deformation of the roller, along with forces due to ‘steering’ on a complex geometry will cause the elastomeric sheath to quickly migrate to one side or the other of the core.  Increasing the surface area of the core (by adding features such as ribs or knurl on the surface of the core where the elastomer will attach), and using a proper primer and adhesive will probably help significantly.  This is what is done with the molded rollers.