[2.2] [WIP] [Finder] Adding native finders implementations #4061

jfsimon · 2012-04-22T10:13:00Z

Work in progress...

Bug fix: no
Feature addition: no
Backwards compatibility break: no
Symfony2 tests pass: yes
Fixes the following tickets: #4031

This PR intends to add native finders implementation based on shell command execution.
Planned support concerns:

GNU find command -> done
MS FINDSTR command

jalliot · 2012-04-22T10:59:58Z

src/Symfony/Component/Finder/Adapter/AdapterInterface.php

+     *
+     * @return bool
+     */
+    function isValid();


What about isSupported?

You're right... i keep it in mind for a future cleanup.

fabpot · 2012-05-15T06:19:50Z

@jfsimon What's the status of this PR?

jfsimon · 2012-05-15T06:43:34Z

@fabpot 2 features missing for the GNU find adapter: sorting result with sort command and excluding directories; 1 bug (even if tests pass, which let me thing it needs more tests): regex matching is done on full path, not basename. Then I'll need to work on MS FINDSTR command adapter (I talked to Pierre Couzy, and he's OK to help when he will have time to). I'll try to push the sort and directory excluding features this week.

jalliot · 2012-05-15T09:51:20Z

BTW @jfsimon, in the (quite specific) case where you don't precise filenames or other options but only contains or notContains, you could call grep directly without the find. That would speed things up a bit more :)

fabpot · 2012-06-28T15:20:55Z

@jfsimon Would be nice to be able to include this PR before 2.1.0 beta2. Would you have time to finish the work soon?

liuggio · 2012-06-28T15:23:22Z

src/Symfony/Component/Finder/Adapter/AdapterInterface.php

+    function searchInDirectory($dir);
+
+    /**
+     * Tests adapter support for current plateform.


platform... is it a typo?

jfsimon · 2012-06-29T11:07:19Z

@fabpot I'd say next week for GNU part with some help from @smaftoul.

pborreli · 2012-06-29T11:31:07Z

src/Symfony/Component/Finder/Command.php

+    public function end()
+    {
+        if (null === $this->parent) {
+            var_dump($this);


maybe var_export inside exception instead if you really need it

this is here for debugging, not for production... removed

jfsimon · 2012-07-10T08:20:44Z

It seems that I need to perform some benchmarks as find may not be so fast :/

jfsimon · 2012-07-10T16:51:19Z

@fabpot @stof do you think I can add benchmark scripts inside the component, or should I create a new repository for that?

fabpot · 2012-07-10T16:57:05Z

Then benchmark scripts won't be part of the repository in the end, so you should create a new repo for that.

jfsimon · 2012-07-13T17:57:03Z

@fabpot @smaftoul Benchmark is ready (more cases to come): https://github.com/jfsimon/symfony-finder-benchmark
I'm glad to see that gnu_find adapter is really faster than the php one!

stof · 2012-07-13T19:17:20Z

@jfsimon could you make a gist with the result of the benchmark ? I think many people will be lazy to run it themselves when looking at this ticket, and people using windows will probably be unable to run it at all :)

jfsimon · 2012-07-13T21:37:50Z

First results: https://gist.github.com/3107676

jfsimon · 2012-08-01T07:26:21Z

Sorry, I forgot [Finder] tag in 3 commits message... is it fixable?

stof · 2012-08-01T08:58:28Z

@jfsimon you can edit the commit message whne doing an interactive rebase.

and btw, you will need to do a rebase anyway: the PR conflicts with master

stof · 2012-08-01T19:32:31Z

src/Symfony/Component/Finder/Finder.php

+    }
+
+    /**
+     * Returns reegistered adapters ordered by priority without extra informations.


two of them ^^

I just see the double e

information without s

jfsimon · 2012-08-01T19:55:02Z

It seems that I'm an idiot, I used contains instead of name.
Real bench is here: https://gist.github.com/3230139
@fabpot sorry for the mess, I should go to bed :/
Results are still convincing!

stof · 2012-08-01T20:04:42Z

They are, but the regex are not faster than glob anymore in these results

travisbot · 2012-08-01T21:12:08Z

This pull request fails (merged aa84e2d into b1618d2).

jfsimon · 2012-08-01T21:17:25Z

@travisbot you failed, not me!

jfsimon · 2012-08-01T21:18:28Z

Anyone to launch benchmark with php 5.4?
https://github.com/jfsimon/symfony-finder-benchmark

lyrixx · 2012-08-01T22:25:08Z

Bench with php 5.4.5
https://gist.github.com/3231244

travisbot · 2012-08-02T05:20:55Z

This pull request passes (merged 5fc01f2 into b1618d2).

travisbot · 2012-08-03T06:40:26Z

This pull request passes (merged ef0ebb8 into b1618d2).

jfsimon · 2012-08-03T06:57:32Z

@fabpot gnu_find adapter is now complete (IMHO). Should I start implementing windows one in this PR or wait for approval/merge and open another one for new adapter? It may takes a while...

stof · 2012-08-03T07:36:07Z

As the Windows implementation could take a while, I would vote for a separate PR so that the GNU adapter can be merged without waiting the windows one.
But this PR will not be merged until the 2.1 stable release as it is scheduled for 2.2

lyrixx · 2012-08-03T09:09:56Z

👍 with @stof. There is no benefits to wait the windows adapter.

jfsimon · 2012-10-03T16:23:16Z

@fabpot Do you need something to be done before merging?

fabpot · 2012-10-03T17:06:34Z

@jfsimon I hadn't had a look at this PR since a long time as it has "WIP" in the title. Can we talk about it tomorrow?

jfsimon · 2012-10-03T17:22:13Z

@fabpot with pleasure :)

sun · 2012-11-07T09:17:32Z

Based on my findings, the actual cause for Finder's poor PHP adapter performance seems to be that it is using Iterators instead of RecursiveIterators. This causes the RecursiveIteratorIterator to perform a full (as in total) recursive filesystem scan without limitation, and Finder filters out unwanted items only afterwards.

By replacing the FilterIterators with RecursiveFilterIterator implementations, the total time for a limited filesystem scan can be easily reduced (cut in half, in my cases), especially when applying directory name or depth filters.

What I'm trying to say is — I'm not sure whether the abstraction into shell adapters is actually needed. It looks like the actual culprit are the iterator implementations, and fixing those could get the PHP adapter fairly close to the numbers of the shell commands — whereas it is easily possible that the additional abstraction into adapters unnecessarily cripples performance again.

Are there any plans to re-implement Finder with Recursive*Iterators?

(of course, that would be a major API break)

FWIW, here are the results of my investigation.

fabpot · 2012-11-07T09:26:26Z

@sun: Thanks for the information. That's very interesting. IIUC, the major API break would be for the iterators, not for the public API (the one in the main Finder class), right? If that's the case, I'm +1 for the refactoring. Would you like to help on this refactoring?

jalliot reviewed Apr 22, 2012
View reviewed changes

liuggio reviewed Jun 28, 2012
View reviewed changes

pborreli reviewed Jun 29, 2012
View reviewed changes

fabpot mentioned this pull request Jul 10, 2012

[RFC] Adding native finders implementations #4031

Closed

stof reviewed Aug 1, 2012
View reviewed changes

[Finder] Fixed typos, CS and small errors.

aa84e2d

jfsimon added 2 commits August 2, 2012 07:03

[Finder] Fixed namespace.

46cc2d4

[Finder] Changed method name for gnu find adapter.

5fc01f2

[Finder] Fixed gnu find regex name filtering problems.

ef0ebb8

fabpot closed this in bb12295 Oct 4, 2012

fabpot mentioned this pull request Nov 9, 2012

Finder slowness #5951

Closed

leofeyer mentioned this pull request May 10, 2013

listFiles must honor globbing better filicious/core#36

Open

Uh oh!

[2.2] [WIP] [Finder] Adding native finders implementations #4061

[2.2] [WIP] [Finder] Adding native finders implementations #4061

Uh oh!

Conversation

jfsimon commented Apr 22, 2012

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fabpot commented May 15, 2012

Uh oh!

jfsimon commented May 15, 2012

Uh oh!

jalliot commented May 15, 2012

Uh oh!

fabpot commented Jun 28, 2012

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfsimon commented Jun 29, 2012

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfsimon commented Jul 10, 2012

Uh oh!

jfsimon commented Jul 10, 2012

Uh oh!

fabpot commented Jul 10, 2012

Uh oh!

jfsimon commented Jul 13, 2012

Uh oh!

stof commented Jul 13, 2012

Uh oh!

jfsimon commented Jul 13, 2012

Uh oh!

jfsimon commented Aug 1, 2012

Uh oh!

stof commented Aug 1, 2012

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfsimon commented Aug 1, 2012

Uh oh!

stof commented Aug 1, 2012

Uh oh!

travisbot commented Aug 1, 2012

Uh oh!

jfsimon commented Aug 1, 2012

Uh oh!

jfsimon commented Aug 1, 2012

Uh oh!

lyrixx commented Aug 1, 2012

Uh oh!

travisbot commented Aug 2, 2012

Uh oh!

travisbot commented Aug 3, 2012

Uh oh!

jfsimon commented Aug 3, 2012

Uh oh!

stof commented Aug 3, 2012

Uh oh!

lyrixx commented Aug 3, 2012

Uh oh!

jfsimon commented Oct 3, 2012

Uh oh!

fabpot commented Oct 3, 2012