UgN dZddlZddlZddlmZddlmZddlm Z ddl m Z m Z m Z ddlmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZdd lmZddlmZm Z ddl!m"Z"m#Z#edd\Z$Z%ee$e%d\Z$Z%e&e$Z$gdZ'dhdej(DzZ)dSdZ*ej+,dedZ-dZ.ej+,dge#e"dZ/dZ0ej+,d e'ej+,d!ed"Z1d#Z2ej+,d$d%d&Z3d'Z4d(Z5d)Z6d*Z7ej+,d+d,d-gd.Z8ej+,d/e#d0Z9ej+,d1e'd2Z:d3Z;d4Zgej>d7ggfd!d6id7d8gd8d7ggfid7d8gd9d:ggfgd;Z?ej+,d/e#d<Z@ej+,d/e#d=ZAd>ZBd?ZCd@ZDej+,dAdBdCgej+,dDddEgdFZEdGZFdHZGej+,dIdJdKgdLZHej+,dMdNdOgdPZIej+,dQd,d-gdRZJdS)TzF Tests for HDBSCAN clustering algorithm Based on the DBSCAN test code N)stats)distance)HDBSCAN)CONDENSED_dtype_condense_tree _do_labelling)_OUTLIER_ENCODING) make_blobs)fowlkes_mallows_score)_VALID_METRICSeuclidean_distances)BallTreeKDTree)StandardScaler)shuffle)assert_allcloseassert_array_equal)CSC_CONTAINERSCSR_CONTAINERS ) n_samples random_state)r)kd_tree ball_treebruteautoc$h|] \}}|dS)label).0_outs a/var/www/surfInsights/venv3-11/lib/python3.11/site-packages/sklearn/cluster/tests/test_hdbscan.py r'&s KKKvq#c'lKKKGz?ctt|tz }|dksJt|t|ksJdS)N)lenset OUTLIER_SETr y)labels threshold n_clusterss r&check_label_qualityr3)sHS[[;.//J ????  + +i 7 7 7 7 7 7r( outlier_typectjtjd|}ddd|}t|d}t|d}t}|dg|d<||g|d<t |}|j|k \}t|ddg||j | \}t|ddgttddttd d z} t || } t| j|j| d S) O Tests if np.inf and np.nan data are each treated as special outliers. )infinitemissingc||kSNr"xr/s r&z#test_outlier_data..9s ar(c*tj|Sr:)npisnanr;s r&r=z#test_outlier_data..:s r(r!probrrN)r?infnanr Xcopyrfitlabels_nonzerorprobabilities_listrange) r4outlier prob_checkr!rA X_outliermodelmissing_labels_idxmissing_probs_idx clean_indices clean_models r&test_outlier_datarW/ss F6G ('++J l +G 4E \ *6 2DIQ>)Aq6222&Ju';TBBKKMM(1a&111q!%%U1c]](;(;;M))-- - 899K{*EM-,HIIIIIr(c^tt}|}tdd|}t ||t |d}tjt|5tddtdddn #1swxYwYd}d|d <d |d <tjt|5td |ddddS#1swxYwYdS) zy Tests that HDBSCAN works with precomputed distance matrices, and throws the appropriate errors when needed. precomputedT)metricrHz*The precomputed distance matrix.*has shapematchNz'The precomputed distance matrix.*valuesr)rrBrB)rBrrZ) r rGrHr fit_predictrr3pytestraises ValueError)D D_originalr0msgs r&test_hdbscan_distance_matrixreOs AAJ M 5 5 5 A A! D DFAz""" 7C z - - -@@}4000<)ABB*QC000Ss__s+=@S+S'T'TTUUI))-- ) 455K00l0KKL|VI%677777r(ctddtjtjdit}t |dS)z4 Tests that HDBSCAN using `BallTree` works. r}rxrB)rZrN)rr?rrGrr^r3rrs r&!test_hdbscan_best_balltree_metricrsVC1D1D+Ek!nn r(ctttdz t}t |t sJdS)z Tests that HDBSCAN correctly does not generate a valid cluster when the `min_cluster_size` is too large for the data. rBmin_cluster_sizeN)rr,rGr^r-issubsetr.rrs r&test_hdbscan_no_clustersrsR c!ffqj 1 1 1 = =a @ @F v;;   , ,,,,,,r(c8tdttdD]u}t|t}d|D}t|dkr,t jt j||ksJvdS)zb Test that the smallest non-noise cluster has at least `min_cluster_size` many points rzrBrcg|] }|dk| S)rr")r#r!s r& z1test_hdbscan_min_cluster_size..s@@@ERKKuKKKr(rN)rNr,rGrr^r?minbincount)rr0 true_labelss r&test_hdbscan_min_cluster_sizers "!SVVQ//HH*:;;;GGJJ@@&@@@ {  q 6"+k22337GGGGG HHr(ctj}t|t}t |dS)zA Tests that HDBSCAN works when passed a callable metric. r]N)r euclideanrr^rGr3)rZr0s r&test_hdbscan_callable_metricrs> F F # # # / / 2 2Fr(treerrctd|}d}tjt|5|t ddddS#1swxYwYdS)z Tests that HDBSCAN correctly raises an error when passing precomputed data while requesting a tree-based algorithm. rYrZrwz%precomputed is not a valid metric forr[N)rr_r`rarIrG)rrrds r&"test_hdbscan_precomputed_non_bruters $ 7 7 7C 1C z - - -  sAAA csr_containercttj}t ||t}|}t|j}t ||tjdftj dffD]\}}t}||d<t|j}t ||dt|dksJ|}||d<t|j}t ||d}tj t|5tdd |d d d d S#1swxYwYd S) z Tests that HDBSCAN works correctly when passing sparse feature data. Evaluates correctness by comparing against the same data passed as a dense array. r7r8rrrr!z4Sparse data matrices only support algorithm `brute`.r[rrrN)rrIrGrJr3rHrr?rErFr r_r`ra) r dense_labels _X_sparseX_sparse sparse_labels outlier_valr4X_denserds r&test_hdbscan_sparser s99==##+L %%% a  I~~HIIMM(++3M|]333(*vz&:RVY>##$ h//7 <7777 @C z - - -II{k:::>>xHHHIIIIIIIIIIIIIIIIIIs%GG G rwcddg}tdd|d\}}td|}t||j|jD],\}}}t ||d d t ||d d -t|dtjd t}|jjddksJ|jjddksJd S)zj Tests that HDBSCAN centers are calculated and stored properly, and are accurate to the data. )riri)@rirr)rrcenters cluster_stdboth) store_centersrBg?)rtolatol)rwrrN) r rrIzip centroids_medoids_rrGr)rwrHr$rcentercentroidmedoids r&test_hdbscan_centersr-s  :&G 1gSV W W WDAq  ' ' ' + +A . .C$'$N$N;; &qt<<<<QT::::: 6AGAJ    c!ff >  "a ' ' ' ' < a A % % % % % %r(c tjd}|dd}t dddd|}tj|d \}}t|dksJ||d kd ksJt dd ddd |}tj|d \}}t|dksJ||d kdksJdS)zS Tests that HDBSCAN single-cluster selection with epsilon works correctly. rrzrCrieomT)rcluster_selection_epsiloncluster_selection_methodallow_single_cluster) return_countsrg ףp= ?r)rrrrrwN)r?random RandomStaterandrr^uniquer,)rng no_structurer0 unique_labelscountss r&.test_hdbscan_allow_single_cluster_with_epsilonrCs; )   " "C88C##L "%!&!  k,  IfDAAAM6 }   " " " " -2% & + + + +"&!&!  k,  IfDAAAM6 }   " " " " -2% &! + + + + + +r(c2ddgddgddgddgg}td|gdd\}}t|j}t t |t d |vz }|d ksJt||d kd S) z Validate that HDBSCAN can properly cluster this difficult synthetic dataset. Note that DBSCAN fails on this (see HDBSCAN plotting example) g333333g333333?r+i)皙?gffffff?皙?rr)rrrrrr)N)r rrIrJr,r-intr )rrGr/r0r2s r&test_hdbscan_better_than_dbscanrds u~t}q!fq"g>G +++    DAq YY]]1   %FS[[!!Cf $5$55J ????&!$$t++++r(z kwargs, XrYrBrzr+rcHtdddi||dS)zo Tests that HDBSCAN works correctly for array-likes and precomputed inputs with non-finite points. min_samplesrBNr")rrI)rGkwargss r&test_hdbscan_usable_inputsrxs1 $$$V$$((+++++r(c|tjd}d}tjt|5t d|ddddS#1swxYwYdS)zd Tests that HDBSCAN raises the correct error when there are too few non-zero distances. )rrz#There exists points with fewer thanr[rYr]N)r?zerosr_r`rarrIrrGrds r&-test_hdbscan_sparse_distances_too_few_nonzerors  bhx(())A /C z - - ---}%%%))!,,,------------------s$A,,A03A0c6tjd}d|ddddf<d|ddddf<||jz}||}d}tjt |5t d |ddddS#1swxYwYdS) zu Tests that HDBSCAN raises the correct error when the distance matrix has multiple connected components. )rrBNrCz2HDBSCAN cannot be perfomed on a disconnected graphr[rYr])r?rTr_r`rarrIrs r&0test_hdbscan_sparse_distances_disconnected_graphrs AAbqb"1"fIAabb"##gJ ACA aA >C z - - ---}%%%))!,,,------------------s$BBBcd}d}tjt|5td|t dddn #1swxYwYtjt|5td|t dddn #1swxYwYt ttj ttj z }t|dkretjt|5td|dt ddddS#1swxYwYdSdS) zR Tests that HDBSCAN correctly raises an error for invalid metric choices. c|Sr:r")r<s r&r=z2test_hdbscan_tree_invalid_metric..sr(zV.* is not a valid metric for a .*-based algorithm\. Please select a different metric\.r[r)rwrZNrr) r_r`rarrIrGrMr-rrrr,)metric_callablerdmetrics_not_kds r& test_hdbscan_tree_invalid_metricrsG"kO  z - - -DD)O<<<@@CCCDDDDDDDDDDDDDDD z - - -FF+o>>>BB1EEEFFFFFFFFFFFFFFF #h455FQ ]:S 1 1 1 J J iq0A B B B F Fq I I I J J J J J J J J J J J J J J J J J Js5*AAA=*B33B7:B7,0E))E-0E-ctttdz}d}tjt |5|tddddS#1swxYwYdS)zx Tests that HDBSCAN correctly raises an error when setting `min_samples` larger than the number of samples. rB)rz min_samples (.*) must be at mostr[N)rr,rGr_r`rarI)rrds r&!test_hdbscan_too_many_min_samplesrs c!ffqj ) ) )C -C z - - -  sA++A/2A/c t}tj|d<d}t d}t jt|5||ddddS#1swxYwYdS)zu Tests that HDBSCAN correctly raises an error when providing precomputed distances with `np.nan` values. rz(np.nan values found in precomputed-denserYr]r[N) rGrHr?rFrr_r`rarI)X_nanrdrs r&"test_hdbscan_precomputed_dense_nanrs FFHHE&E$K 4C  ' ' 'C z - - - sA99A=A=rTFepsilonrc0 d}t||ddgddgddgg\} t|}t|j|j}|dz|dz|dzh}|dzd|dzd |dzdi}t |||||  fd tt D fd tt D} tj | j  } t | d S)zR Tests that the `_do_labelling` helper function correctly assigns labels. 0rr)rrrrzr+rrBcondensed_treeclusterscluster_label_maprrcZi|]'}|tj|kdd(S)r)r?where)r#_yr/s r& z+test_labelling_distinct..s3KKKBHQ"W--a03KKKr(c.i|]}||Sr"r")r#rfirst_with_labelr0s r&rz+test_labelling_distinct..s&KKK2v.r23KKKr(N) r rrIr_single_linkage_tree_rrrMr-r? vectorizerr)global_random_seedrrrrGestrrr y_to_labelsaligned_targetrr0r/s @@@r&test_labelling_distinctr sZ I 'F G G   DAq ))--  C# !C4HNA y1}i!m>S2.. . . . . %$aQ:!"# Fw'*4I y>>S2.. . . . . . .r(c|d}tjt|5tdt dddn #1swxYwYd}tjt|5tdt ddddS#1swxYwYdS)Nz`algorithm='kdtree'`has been deprecated in 1.4 and will be renamed to'kd_tree'`in 1.6. To keep the past behaviour, set `algorithm='kd_tree'`.r[kdtreervz`algorithm='balltree'`has been deprecated in 1.4 and will be renamed to'ball_tree'`in 1.6. To keep the past behaviour, set `algorithm='ball_tree'`.balltree)r_rrrrIrG)rds r&1test_hdbscan_warning_on_deprecated_algorithm_namer&s< V m3 / / /++(###''***+++++++++++++++  $ m3 / / /--*%%%))!,,,------------------s#)AAA;)B11B58B5rrrc@tjd}|d}t|}d}t jt |5td||ddddS#1swxYwYdS)zCheck that we raise an error if the centers are requested together with a precomputed input matrix. Non-regression test for: https://github.com/scikit-learn/scikit-learn/issues/27893 r)drzz>Cannot store centers when using a precomputed distance matrix.r[rY)rZrN) r?rrr r_r`rarrI)rrrGX_disterr_msgs r&0test_hdbscan_error_precomputed_and_store_centersr9s )   " "C 8A  # #FNG z 1 1 1OO}MBBBFFvNNNOOOOOOOOOOOOOOOOOOs!%BBB valid_algorrcXtd|tdS)zTest that HDBSCAN works with the "cosine" metric when the algorithm is set to "brute" or "auto". Non-regression test for issue #28631 cosinerN)rr^rG)rs r&*test_hdbscan_cosine_metric_valid_algorithmr!Is+ 8z222>>qAAAAAr( invalid_algoctd|}tjtd5|t ddddS#1swxYwYdS)zTest that HDBSCAN raises an informative error is raised when an unsupported algorithm is used with the "cosine" metric. r rzcosine is not a valid metricr[N)rr_r`rar^rG)r"hdbscans r&,test_hdbscan_cosine_metric_invalid_algorithmr%Ss X>>>G z)G H H HAsAAA)r))K__doc__numpyr?r_scipyr scipy.spatialrsklearn.clusterrsklearn.cluster._hdbscan._treerrr sklearn.cluster._hdbscan.hdbscanr sklearn.datasetsr sklearn.metricsr sklearn.metrics.pairwiser r sklearn.neighborsrrsklearn.preprocessingr sklearn.utilsrsklearn.utils._testingrrsklearn.utils.fixesrrrGr/ fit_transform ALGORITHMSitemsr.r3mark parametrizerWrerprsrrrrrrrrrrrrrrErrrrrrr rrrr!r%r"r(r&r:s(  """"""###### ?>>>>>''''''111111HHHHHHHH........000000!!!!!!FFFFFFFF>>>>>>>>zCb1111wq!!$$$1N""1%% dKK1H1B1H1J1JKKKK 8888 ):;;JJ<;J>5550-/Q/Q./QRR  SR "    ,,>22$$32-,$N 0 0 077888784   --- H H H   )[!9::;:.99II:9IDj11&&21&*,,,B,,,( M "HBHq"&kBFA;-G$H$HI M "aVaV$45 q!fq!f ,,,.99 - -:9 -.99 - -:9 - JJJ0   /$??QH--!/!/.-@?!/H&/&/&/T---&:x*@AA O OBA O'899BB:9B)[)ABBCBr(