How to query Products according to the size?

So I have model Product which must be filtered by color, price, and size. Here are my relationships.

class Product extends Model
{
    public function color()
    {
        return $this->belongsTo('AppColor');
    }

    public function sizes()
    {
        return $this->belongsToMany('AppSize', 'product_size')->withTimestamps();
    }
}

Here is my Size model:

class Size extends Model
{
    public function products()
    {
        return $this->belongsToMany('AppProduct', 'product_size')->withTimestamps();
    }
}

Here is my form:

<form id="filterOptions" method="get" action="{{ URL::current() }}">
    <button type="submit" class="btn-color">Filter</button>

    <div class="clearfix space20"></div>
    <h5>Color</h5>
    <ul class="color-list">
        @foreach($availableColors as $color)
            <li><input type="checkbox" name="color[]" value="{{ $color->id }}"><a href="#"><span class="{{ $color->name }}"></span> {{$color->name_bg}}</a></li>
            {{ $color->name }}
        @endforeach
    </ul>
    <div class="clearfix space20"></div>

    <h5>Price</h5>
    <div id="slider-container"></div>
    <p>
        <span class="{{--pull-right--}} sc-range">
            <input class="pull-left" name="min" type="text" id="min" style="border: 0; color: #333333; font-weight: bold;"/>
            <input class="pull-right" name="max" type="text" id="max" style="border: 0; color: #333333; font-weight: bold;"/>
        </span>
    </p>

    <div class="clearfix space30"></div>
    <h5>Size</h5>
    <ul class="size-list">
        @foreach($availableSizes as $size)
            <li><input type="checkbox" name="size[]" value="{{ $size->id }}">{{ $size->size }}</li>
        @endforeach
    </ul>
</form>

I manage to query the products by price and color without problems. However I can't get it to query when it is a many to many relationship. How can I do that? Here is my code so far:

$minPrice = $request['min'];
$maxPrice = $request['max'];
$colors = $request['color'];
$sizes = $request['size'];

if (count($request->all()) != 0) {
    $query = Product::with(['sizes' => function($query) use($request) {
        $sizeArray = $request->get('size');
        $query->whereIn('size', $sizeArray);
    }]);

    if(isset($minPrice) && isset($maxPrice)) {
        $query->whereBetween('price', array($minPrice, $maxPrice));
    }

    if(isset($colors)) {
        $query->whereIn('color_id', $colors);
    }

    $products = $query->get();
}

Everything works besides the query by sizes. How can I fix this? Please, I can't find a solution to this.