E581 Add use-cluster-ip annotation for ingress resources (#4862) · nginx/kubernetes-ingress@38a03fa · GitHub
[go: up one dir, main page]
Skip to content

Commit 38a03fa

Browse files
Jim RyanADubhlaoich
Jim Ryan
and
ADubhlaoich
authored
Add use-cluster-ip annotation for ingress resources (#4862)
* add use cluter ip annotation * Update docs/content/configuration/ingress-resources/advanced-configuration-with-annotations.md Co-authored-by: Alan Dooley <github@adubhlaoi.ch> Signed-off-by: Jim Ryan <j.ryan@f5.com> * annotation validation * remove duplicate test * small docs fix * go ingress tests * use fqdn to support cross namespace ingresses --------- Signed-off-by: Jim Ryan <j.ryan@f5.com> Co-authored-by: Alan Dooley <github@adubhlaoi.ch>
1 parent 99c9212 commit 38a03fa

File tree

10 files changed

+325
-10
lines changed

10 files changed

+325
-10
lines changed

‎docs/content/configuration/ingress-resources/advanced-configuration-with-annotations.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,7 @@ The table below summarizes the available annotations.
172172
|``nginx.com/health-checks-mandatory`` | N/A | Configures active health checks as mandatory. | ``False`` | [Support for Active Health Checks](https://github.com/nginxinc/kubernetes-ingress/tree/v3.4.0/examples/ingress-resources/health-checks). |
173173
|``nginx.com/health-checks-mandatory-queue`` | N/A | When active health checks are mandatory, creates a queue where incoming requests are temporarily stored while NGINX Plus is checking the health of the endpoints after a configuration reload. | ``0`` | [Support for Active Health Checks](https://github.com/nginxinc/kubernetes-ingress/tree/v3.4.0/examples/ingress-resources/health-checks). |
174174
|``nginx.com/slow-start`` | N/A | Sets the upstream server [slow-start period](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#server-slow-start). By default, slow-start is activated after a server becomes [available](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#passive-health-checks) or [healthy](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/#active-health-checks). To enable slow-start for newly-added servers, configure [mandatory active health checks](https://github.com/nginxinc/kubernetes-ingress/tree/v3.4.0/examples/ingress-resources/health-checks). | ``"0s"`` | |
175+
|``nginx.org/use-cluster-ip`` | N/A | Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like ``lb-method`` and ``next-upstream``) will have no effect, as NGINX Ingress Controller will configure NGINX with only one upstream server that will match the service Cluster IP. | ``False`` | |
175176
{{% /table %}}
176177

177178
### Snippets and Custom Templates

‎docs/content/configuration/virtualserver-and-virtualserverroute-resources.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -349,7 +349,7 @@ tls:
349349
|``name`` | The name of the upstream. Must be a valid DNS label as defined in RFC 1035. For example, ``hello`` and ``upstream-123`` are valid. The name must be unique among all upstreams of the resource. | ``string`` | Yes |
350350
|``service`` | The name of a [service](https://kubernetes.io/docs/concepts/services-networking/service/). The service must belong to the same namespace as the resource. If the service doesn't exist, NGINX will assume the service has zero endpoints and return a ``502`` response for requests for this upstream. For NGINX Plus only, services of type [ExternalName](https://kubernetes.io/docs/concepts/services-networking/service/#externalname) are also supported (check the [prerequisites](https://github.com/nginxinc/kubernetes-ingress/tree/v3.4.0/examples/ingress-resources/externalname-services#prerequisites) ). | ``string`` | Yes |
351351
|``subselector`` | Selects the pods within the service using label keys and values. By default, all pods of the service are selected. Note: the specified labels are expected to be present in the pods when they are created. If the pod labels are updated, the Ingress Controller will not see that change until the number of the pods is changed. | ``map[string]string`` | No |
352-
|``use-cluster-ip`` | Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like ``lb-method`` and ``next-upstream``) will have no effect, as the Ingress Controller will configure NGINX with only one upstream server that will match the service Cluster IP. | ``boolean`` | No |
352+
|``use-cluster-ip`` | Enables using the Cluster IP and port of the service instead of the default behavior of using the IP and port of the pods. When this field is enabled, the fields that configure NGINX behavior related to multiple upstream servers (like ``lb-method`` and ``next-upstream``) will have no effect, as NGINX Ingress Controller will configure NGINX with only one upstream server that will match the service Cluster IP. | ``boolean`` | No |
353353
|``port`` | The port of the service. If the service doesn't define that port, NGINX will assume the service has zero endpoints and return a ``502`` response for requests for this upstream. The port must fall into the range ``1..65535``. | ``uint16`` | Yes |
354354
|``lb-method`` | The load [balancing method](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/#choosing-a-load-balancing-method). To use the round-robin method, specify ``round_robin``. The default is specified in the ``lb-method`` ConfigMap key. | ``string`` | No |
355355
|``fail-timeout`` | The time during which the specified number of unsuccessful attempts to communicate with an upstream server should happen to consider the server unavailable. See the [fail_timeout](https://nginx.org/en/docs/http/ngx_http_upstream_module.html#fail_timeout) parameter of the server directive. The default is set in the ``fail-timeout`` ConfigMap key. | ``string`` | No |

‎docs/content/tutorials/nginx-ingress-istio.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -214,7 +214,7 @@ We can see in the above output that our curl request is sent and received by NGI
214214

215215
By default, for NGINX Ingress Controller, we populate the upstream server addresses with the endpoint IPs of the pods.
216216

217-
When using the new `use-cluster-ip` feature, we will no populate the upstream with the `service` IP address, instead of the endpoint IP addresses.
217+
When using the new `use-cluster-ip` feature, we will now populate the upstream with the `service` IP address, instead of the endpoint IP addresses.
218218

219219
In the 1.11 release, NGINX Ingress controller will only send one host header, depending on how you configure Ingress. By default NGINX Ingress Controller will send `proxy_set_header $host`. If Ingress has been configured with `action.proxy.requestHeaders` this ensures that only one set of headers will be sent to the upstream server. In summary, by setting `action.proxy.requestHeaders` in the `VirtualServer` CRD, NGINX Ingress will only send the specified headers that have been defined.
220220

‎examples/ingress-resources/mergeable-ingress-types/cafe.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,6 @@ apiVersion: v1
5555
kind: Service
5656
metadata:
5757
name: tea-svc
58-
labels:
5958
spec:
6059
ports:
6160
- port: 80

‎internal/configs/annotations.go‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@ const BasicAuthSecretAnnotation = "nginx.org/basic-auth-secret" // #nosec G101
1313
// PathRegexAnnotation is the annotation where the regex location (path) modifier is specified.
1414
const PathRegexAnnotation = "nginx.org/path-regex"
1515

16+
// UseClusterIPAnnotation is the annotation where the use-cluster-ip boolean is specified.
17+
const UseClusterIPAnnotation = "nginx.org/use-cluster-ip"
18+
1619
// AppProtectPolicyAnnotation is where the NGINX App Protect policy is specified
1720
const AppProtectPolicyAnnotation = "appprotect.f5.com/app-protect-policy"
1821

@@ -37,6 +40,7 @@ var masterBlacklist = map[string]bool{
3740
"nginx.com/health-checks": true,
3841
"nginx.com/health-checks-mandatory": true,
3942
"nginx.com/health-checks-mandatory-queue": true,
43+
UseClusterIPAnnotation: true,
4044
}
4145

4246
var minionBlacklist = map[string]bool{
@@ -401,6 +405,14 @@ func parseAnnotations(ingEx *IngressEx, baseCfgParams *ConfigParams, isPlus bool
401405
glog.Errorf("Ingress %s/%s: Invalid value nginx.org/path-regex: got %q. Allowed values: 'case_sensitive', 'case_insensitive', 'exact'", ingEx.Ingress.GetNamespace(), ingEx.Ingress.GetName(), pathRegex)
402406
}
403407
}
408+
409+
if useClusterIP, exists, err := GetMapKeyAsBool(ingEx.Ingress.Annotations, UseClusterIPAnnotation, ingEx.Ingress); exists {
410+
if err != nil {
411+
glog.Error(err)
412+
} else {
413+
cfgParams.UseClusterIP = useClusterIP
414+
}
415+
}
404416
return cfgParams
405417
}
406418

‎internal/configs/config_params.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ type ConfigParams struct {
8383
SlowStart string
8484
SSLRedirect bool
8585
UpstreamZoneSize string
86+
UseClusterIP bool
8687
VariablesHashBucketSize uint64
8788
VariablesHashMaxSize uint64
8889

‎internal/configs/ingress.go‎

Lines changed: 20 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -511,21 +511,34 @@ func createUpstream(ingEx *IngressEx, name string, backend *networking.IngressBa
511511
endps = []string{}
512512
}
513513

514-
for _, endp := range endps {
514+
if cfg.UseClusterIP {
515+
fqdn := fmt.Sprintf("%s.%s.svc.cluster.local:%d", backend.Service.Name, ingEx.Ingress.Namespace, backend.Service.Port.Number)
515516
upsServers = append(upsServers, version1.UpstreamServer{
516-
Address: endp,
517+
Address: fqdn,
517518
MaxFails: cfg.MaxFails,
518519
MaxConns: cfg.MaxConns,
519520
FailTimeout: cfg.FailTimeout,
520521
SlowStart: cfg.SlowStart,
521522
Resolve: isExternalNameSvc,
522523
})
523-
}
524-
if len(upsServers) > 0 {
525-
sort.Slice(upsServers, func(i, j int) bool {
526-
return upsServers[i].Address < upsServers[j].Address
527-
})
528524
ups.UpstreamServers = upsServers
525+
} else {
526+
for _, endp := range endps {
527+
upsServers = append(upsServers, version1.UpstreamServer{
528+
Address: endp,
529+
MaxFails: cfg.MaxFails,
530+
MaxConns: cfg.MaxConns,
531+
FailTimeout: cfg.FailTimeout,
532+
SlowStart: cfg.SlowStart,
533+
Resolve: isExternalNameSvc,
534+
})
535+
}
536+
if len(upsServers) > 0 {
537+
sort.Slice(upsServers, func(i, j int) bool {
538+
return upsServers[i].Address < upsServers[j].Address
539+
})
540+
ups.UpstreamServers = upsServers
541+
}
529542
}
530543
}
531544

‎internal/configs/ingress_test.go‎

Lines changed: 247 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -663,6 +663,253 @@ func TestGenerateNginxCfgForMergeableIngressesForBasicAuth(t *testing.T) {
663663
}
664664
}
665665

666+
func TestGenerateNginxCfgForMergeableIngressesWithUseClusterIP(t *testing.T) {
667+
t.Parallel()
668+
mergeableIngresses := createMergeableCafeIngress()
669+
mergeableIngresses.Minions[0].Ingress.Annotations["nginx.org/use-cluster-ip"] = "true"
670+
671+
isPlus := false
672+
673+
expected := createExpectedConfigForMergeableCafeIngressWithUseClusterIP()
674+
configParams := NewDefaultConfigParams(isPlus)
675+
676+
result, warnings := generateNginxCfgForMergeableIngresses(NginxCfgParams{
677+
mergeableIngs: mergeableIngresses,
678+
apResources: nil,
679+
dosResource: nil,
680+
baseCfgParams: configParams,
681+
isPlus: isPlus,
682+
isResolverConfigured: false,
683+
staticParams: &StaticConfigParams{},
684+
isWildcardEnabled: false,
685+
})
686+
687+
if diff := cmp.Diff(expected, result); diff != "" {
688+
t.Errorf("generateNginxCfgForMergeableIngresses() returned unexpected result (-want +got):\n%s", diff)
689+
}
690+
if len(warnings) != 0 {
691+
t.Errorf("generateNginxCfgForMergeableIngresses() returned warnings: %v", warnings)
692+
}
693+
}
694+
695+
func createExpectedConfigForMergeableCafeIngressWithUseClusterIP() version1.IngressNginxConfig {
696+
upstreamZoneSize := "256k"
697+
coffeeUpstream := version1.Upstream{
698+
Name: "default-cafe-ingress-coffee-minion-cafe.example.com-coffee-svc-80",
699+
LBMethod: "random two least_conn",
700+
UpstreamZoneSize: upstreamZoneSize,
701+
UpstreamServers: []version1.UpstreamServer{
702+
{
703+
Address: "coffee-svc.default.svc.cluster.local:80",
704+
MaxFails: 1,
705+
MaxConns: 0,
706+
FailTimeout: "10s",
707+
},
708+
},
709+
}
710+
teaUpstream := version1.Upstream{
711+
Name: "default-cafe-ingress-tea-minion-cafe.example.com-tea-svc-80",
712+
LBMethod: "random two least_conn",
713+
UpstreamZoneSize: upstreamZoneSize,
714+
UpstreamServers: []version1.UpstreamServer{
715+
{
716+
Address: "10.0.0.2:80",
717+
MaxFails: 1,
718+
MaxConns: 0,
719+
FailTimeout: "10s",
720+
},
721+
},
722+
}
723+
expected := version1.IngressNginxConfig{
724+
Upstreams: []version1.Upstream{
725+
coffeeUpstream,
726+
teaUpstream,
727+
},
728+
Servers: []version1.Server{
729+
{
730+
Name: "cafe.example.com",
731+
ServerTokens: "on",
732+
Locations: []version1.Locat 28B5 ion{
733+
{
734+
Path: "/coffee",
735+
ServiceName: "coffee-svc",
736+
Upstream: coffeeUpstream,
737+
ProxyConnectTimeout: "60s",
738+
ProxyReadTimeout: "60s",
739+
ProxySendTimeout: "60s",
740+
ClientMaxBodySize: "1m",
741+
ProxyBuffering: true,
742+
MinionIngress: &version1.Ingress{
743+
Name: "cafe-ingress-coffee-minion",
744+
Namespace: "default",
745+
Annotations: map[string]string{
746+
"kubernetes.io/ingress.class": "nginx",
747+
"nginx.org/mergeable-ingress-type": "minion",
748+
"nginx.org/use-cluster-ip": "true",
749+
},
750+
},
751+
ProxySSLName: "coffee-svc.default.svc",
752+
},
753+
{
754+
Path: "/tea",
755+
ServiceName: "tea-svc",
756+
Upstream: teaUpstream,
757+
ProxyConnectTimeout: "60s",
758+
ProxyReadTimeout: "60s",
759+
ProxySendTimeout: "60s",
760+
ClientMaxBodySize: "1m",
761+
ProxyBuffering: true,
762+
MinionIngress: &version1.Ingress{
763+
Name: "cafe-ingress-tea-minion",
764+
Namespace: "default",
765+
Annotations: map[string]string{
766+
"kubernetes.io/ingress.class": "nginx",
767+
"nginx.org/mergeable-ingress-type": "minion",
768+
},
769+
},
770+
ProxySSLName: "tea-svc.default.svc",
771+
},
772+
},
773+
SSL: true,
774+
SSLCertificate: "/etc/nginx/secrets/default-cafe-secret",
775+
SSLCertificateKey: "/etc/nginx/secrets/default-cafe-secret",
776+
StatusZone: "cafe.example.com",
777+
HSTSMaxAge: 2592000,
778+
Ports: []int{80},
779+
SSLPorts: []int{443},
780+
SSLRedirect: true,
781+
HealthChecks: make(map[string]version1.HealthCheck),
782+
},
783+
},
784+
Ingress: version1.Ingress{
785+
Name: "cafe-ingress-master",
786+
Namespace: "default",
787+
Annotations: map[string]string{
788+
"kubernetes.io/ingress.class": "nginx",
789+
"nginx.org/mergeable-ingress-type": "master",
790+
},
791+
},
792+
}
793+
794+
return expected
795+
}
796+
797+
func createExpectedConfigForCafeIngressWithUseClusterIP() version1.IngressNginxConfig {
798+
upstreamZoneSize := "256k"
799+
800+
coffeeUpstream := version1.Upstream{
801+
Name: "default-cafe-ingress-cafe.example.com-coffee-svc-80",
802+
LBMethod: "random two least_conn",
803+
UpstreamZoneSize: upstreamZoneSize,
804+
UpstreamServers: []version1.UpstreamServer{
805+
{
806+
Address: "coffee-svc.default.svc.cluster.local:80",
807+
MaxFails: 1,
808+
MaxConns: 0,
809+
FailTimeout: "10s",
810+
},
811+
},
812+
}
813+
814+
teaUpstream := version1.Upstream{
815+
Name: "default-cafe-ingress-cafe.example.com-tea-svc-80",
816+
LBMethod: "random two least_conn",
817+
UpstreamZoneSize: upstreamZoneSize,
818+
UpstreamServers: []version1.UpstreamServer{
819+
{
820+
Address: "tea-svc.default.svc.cluster.local:80",
821+
MaxFails: 1,
822+
MaxConns: 0,
823+
FailTimeout: "10s",
824+
},
825+
},
826+
}
827+
828+
expected := version1.IngressNginxConfig{
829+
Upstreams: []version1.Upstream{
830+
coffeeUpstream,
831+
teaUpstream,
832+
},
833+
Servers: []version1.Server{
834+
{
835+
Name: "cafe.example.com",
836+
ServerTokens: "on",
837+
Locations: []version1.Location{
838+
{
839+
Path: "/coffee",
840+
ServiceName: "coffee-svc",
841+
Upstream: coffeeUpstream,
842+
ProxyConnectTimeout: "60s",
843+
ProxyReadTimeout: "60s",
844+
ProxySendTimeout: "60s",
845+
ClientMaxBodySize: "1m",
846+
ProxyBuffering: true,
847+
ProxySSLName: "coffee-svc.default.svc",
848+
},
849+
{
850+
Path: "/tea",
851+
ServiceName: "tea-svc",
852+
Upstream: teaUpstream,
853+
ProxyConnectTimeout: "60s",
854+
ProxyReadTimeout: "60s",
855+
ProxySendTimeout: "60s",
856+
ClientMaxBodySize: "1m",
857+
ProxyBuffering: true,
858+
ProxySSLName: "tea-svc.default.svc",
859+
},
860+
},
861+
SSL: true,
862+
SSLCertificate: "/etc/nginx/secrets/default-cafe-secret",
863+
SSLCertificateKey: "/etc/nginx/secrets/default-cafe-secret",
864+
StatusZone: "cafe.example.com",
865+
HSTSMaxAge: 2592000,
866+
Ports: []int{80},
867+
SSLPorts: []int{443},
868+
SSLRedirect: true,
869+
HealthChecks: make(map[string]version1.HealthCheck),
870+
},
871+
},
872+
Ingress: version1.Ingress{
873+
Name: "cafe-ingress",
874+
Namespace: "default",
875+
Annotations: map[string]string{
876+
"kubernetes.io/ingress.class": "nginx",
877+
"nginx.org/use-cluster-ip": "true",
878+
},
879+
},
880+
}
881+
return expected
882+
}
883+
884+
func TestGenerateNginxCfgWithUseClusterIP(t *testing.T) {
885+
t.Parallel()
886+
cafeIngressEx := createCafeIngressEx()
887+
cafeIngressEx.Ingress.Annotations["nginx.org/use-cluster-ip"] = "true"
888+
isPlus := false
889+
configParams := NewDefaultConfigParams(isPlus)
890+
891+
expected := createExpectedConfigForCafeIngressWithUseClusterIP()
892+
893+
result, warnings := generateNginxCfg(NginxCfgParams{
894+
staticParams: &StaticConfigParams{},
895+
ingEx: &cafeIngressEx,
896+
apResources: nil,
897+
dosResource: nil,
898+
isMinion: false,
899+
isPlus: false,
900+
baseCfgParams: configParams,
901+
isResolverConfigured: false,
902+
isWildcardEnabled: false,
903+
})
904+
905+
if diff := cmp.Diff(expected, result); diff != "" {
906+
t.Errorf("generateNginxCfg() returned unexpected result (-want +got):\n%s", diff)
907+
}
908+
if len(warnings) != 0 {
909+
t.Errorf("generateNginxCfg() returned warnings: %v", warnings)
910+
}
911+
}
912+
666913
func createMergeableCafeIngress() *MergeableIngresses {
667914
master := networking.Ingress{
668915
ObjectMeta: meta_v1.ObjectMeta{

‎internal/k8s/validation.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ const (
6969
rewritesAnnotation = "nginx.org/rewrites"
7070
stickyCookieServicesAnnotation = "nginx.com/sticky-cookie-services"
7171
pathRegexAnnotation = "nginx.org/path-regex"
72+
useClusterIPAnnotation = "nginx.org/use-cluster-ip"
7273
)
7374

7475
const (
@@ -332,6 +333,9 @@ var (
332333
pathRegexAnnotation: {
333334
validatePathRegex,
334335
},
336+
useClusterIPAnnotation: {
337+
validateBoolAnnotation,
338+
},
335339
}
336340
annotationNames = sortedAnnotationNames(annotationValidations)
337341
)

0 commit comments

Comments
 (0)
0