On Collision Enter Method Stuck With Slow Motion Time

So I’ve been trying to get time slow down (and speed ups too) working on a project I’m doing but I’ve hit a wall. If time has been slowed, the on collision enter method I’m using below (for in this case just testing a bouncing sphere) gets stuck (I suspect it has to do with when the object has been slowed the method thinks the object is colliding with it again). Here’s the code below:

using UnityEngine;
using System.Collections;

public class testTime2 : MonoBehaviour {

    private Rigidbody rb;
    private bool keyPressed; 

    void Start() {
        rb = GetComponent<Rigidbody>();
    }

    void FixedUpdate()
    {
        if (Input.GetKey(KeyCode.Space))
        {
            keyPressed = true;
            rb.velocity = rb.velocity * Time.deltaTime * 0.2f;
        }
        else
            keyPressed = false;
    } 

    void OnCollisionEnter() // method'stuck' with a slow object velocity
    {
        rb.AddForce(transform.up * 7f, ForceMode.Impulse);
        // my attempt to workaround it (doesn't work)
        if (keyPressed)
            rb.velocity = rb.velocity * Time.deltaTime * 1.0f; 
        else
            rb.velocity = rb.velocity * Time.deltaTime * 0.2f;
    }
}

I thought maybe a workaround of temporarily pushing the velocity back up would fix it, but I had no luck. Visually it looks like this:

https://drive.google.com/file/d/0B1wVSPywFDMaMmNCWWN3SGcyYkE/view?usp=sharing

[Here I press the spacebar as it reaches the floor for a slowdown, the spehere position keeps popping, then I let go of the spacebar for velocity to return to normal (which fixes it)]

I know there’s a couple of other posts about slowing down time on specific objects but I was hoping that a simple velocity change would do it (in this case it isn’t cutting it though). Any ideas on how I could get this to work? I know there’s Time.scale but that’ll effect all rigidbodies, here I’m doing it on a case by case basis.

If you are bouncing the ball just on planar or rectangular surfaces i would recommend you to implement custom bouncing ball physics.
Someting like:

using UnityEngine;
using System.Collections;

public class Test : MonoBehaviour
{

    private Rigidbody rb;
    private bool enter = false;
    private bool exit = false;

    public Vector3 Velocity = Vector3.zero;
    private Vector3 Gravity = new Vector3(0.0f, -10.0f, .0f);
    private float mass = 1.0f;
    private Vector3 Impulse = new Vector3(0.0f, 500.0f, 0.0f);
    private Vector3 acceleration;
    private Vector3 TempVelocity;
    public float timeMultipler = 1.0f;

    void Start()
    {
        rb = GetComponent<Rigidbody>();
    }

    void FixedUpdate()
    {
        if (enter)
        {
            acceleration = Gravity + Impulse / mass;
            enter = false;
        }
        else
        {
            acceleration = Gravity;
        }

        Velocity += acceleration * Time.fixedDeltaTime * timeMultipler;
        transform.position += Velocity * Time.fixedDeltaTime * timeMultipler;
    }

    void OnTriggerEnter()
    {
        enter = true;
        TempVelocity = Velocity; // velocity back up
    }
    void OnTriggerStay() 
    {
        enter = true;
    }
    void OnTriggerExit()
    {
        exit = true;
        Velocity = new Vector3(TempVelocity.x, -TempVelocity.y * 0.9899999f, TempVelocity.z); //magic number for perfect bounciness 0.9899999f // you can modify if you need to
    }
}

For this script to work, you need to make your rigidbody kinematic and make your sphere collider trigger.
You can modify timeMultipler for slowing down. Also you can modify x and z velocity components.